Performance Management in Competitive Distributed Web Search

Distributed heterogeneous search environments are an emerging phenomenon in Web search. They can be viewed as a federation of independently controlled metasearchers and many specialised search engines. Specialised search engines provide focused search services in a specific domain (e.g. a particular topic). Metasearchers help to process user queries effectively and efficiently by distributing them only to the most suitable search engines for each query. Compared to the traditional search engines like Google or AltaVista, specialised search engines (together) provide access to arguably much larger volumes of high-quality information resources, frequently called the “deep” or “invisible” Web [10]. We envisage that such heterogeneous environments will become more popular and influential. Previous research has mainly targeted performance of such environments from the user’s perspective (e.g. improved quality of search results). However, a provider of search services is more interested in parameters like the number of queries processed versus the amount of resources used. Performance optimisation of search engines from the service provider’s point of view is the focus of our work. An important factor that affects the search engine performance in heterogeneous search environments is competition with other independently controlled search engines. When there are many engines available, users are interested in sending queries to those that would provide the best possible results. Thus, the service offered by one search engine influences queries received by its competitors. Multiple search providers can be viewed as participants in a search services market competing for user queries. We examine the problem of performance-maximising behaviour for non-cooperative specialised search engines in heterogeneous search environments. In particular, we analyse a scenario in which individual search engines compete for queries by choosing which documents to index. We provide game-theoretic analysis of a simplified version of the problem and motivate the use of the concept of “bounded rationality” [9]. Bounded rationality assumes that decision makers are unable to act optimally in the game-theoretic sense due to incomplete information about the environment and/or limited computational resources. We then cast our problem as a reinforcement learning task, where the goal of a specialised search engine is to exploit sub-optimal behaviour of its competitors to improve own performance.